A formal grammar is a set of production rules for interpreting a sequence of characters. Formal grammars may be used in search queries to recognize specific patterns represented by certain tokens and rules. However, tokens often vary among different files and file types. For example, different log files may have different separators for splitting fields, such as ‘,’, ‘#’, ‘%’, etc., or may use different special character sets. The formal grammar may have to be adjusted in order to accommodate various input strings.